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implications are then identified. Two alternative models are then 
proposed, one of which includes five broad outcome areas (presence, 
participation in the system, accomplishment, contribution, and 
satisfaction) while the other views all outcomes as accomplishments 
and evaluates them for nine accomplishment domains (e.g., cognitive 
and conceptual, attitudes, physical and health, satisfaction, 
adaptive living). A system of indicators for assessing outcomes is 
then proposed. Finally, the following unresolved issues are 
addressed: (1) intended versus unintended outcomes? (2) direct versus 
indirect outcomes? (3) same versus different indicators? (4) category 
specific versus noncategorical indicators? (5) indicators 
differentiated by severity of disability? (6) indicators 
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This paper was prepared to elicit reactions on initial ideas related to definitions, assumptions, and a 
preliminary model of outcomes for children and youth with disabilities. This paper should not be 
quoted or cited at this time. Models and indicators identified herein are tentative and will change. 
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Introduction 



The current report card on American education does not look good. Americas 
public schools continue to experience trouble despite nearly a decade of reform 
activities. Recent reports from commissions, task forces, and public opinion polls 
highlight the continued presence of significant problems in current educational 
practice* Schools and school personnel are criticized for failing to deliver with 
excellence. In addressing these problems, consistent themes in the reports are that 
schools and school personnel will be expected to deliver more than ever before, and 
to be held accountable for the outcomes. 

The current emphasis on educational opportunities and outcomes for students 
in America's schools is said to apply to all students. Yet, policy makers who have 
been identifying goals and considering methods for assessing outcomes for students 
in America's public schools often have failed to consider the significant number of 
students with disabilities. Legislation and social initiatives have increased the 
diversity of students attending schools, including those with limited English 
proficiency, those with different cultural backgrounds, and those with disabilities. 
Included in our schools today are students with visual and hearing impairments, 
emotional disabilities, mental retardation, learning disabilities, physical 
impairments, and severe multiple disabilities. The National Council on Disability 
(1989) has indicated that 

The time has come to ask the same questions for students with 
disabilities that we have been asking about students without disabilities, 
Are they achieving? Are they staying in school? Are they prepared to 
enter the work force when they finish school? Are they going on to 
participate in post secondary education and training? Are they 
prepared for adult life? (p. 2) 

Calls for reform in American education are accompanied by calls to develop 
better measures to assess the broad results of learning and teaching. Work groups, 
advisory boards, and panels are convened to decide how best to assess educational 
outcomes. The desire to assess the results of education and progress toward national 
goals has led to the need for a comprehensive system of outcome indicators. 

Public investment in educational programs for children and youth with 
disabilities is now generating interest in examining educational outcomes for 



NCEO Draft Working Paper 
7/22/91 

children and youth with disabilities. This interest reflects two major aspects of the 
current state of affairs for special education programs. First, special education is at a 
point where it is necessary to move beyond the concern with equal access to 
education. As argued by the National Council on Disability (1989), it is time to 
concentrate on the quality of educational experiences for children and youth with 
disabilities. In order to make judgments about educational quality, we need to have 
indices by which to judge the quality and nature of educational experiences and 
results. A second aspect of the current interest in developing indicators for students 
with disabilities is that general education is proceeding with its own agenda to raise 
expectations for students and to identify outcomes to be reached by "air students. 
General education policy is being established with limited recognition of students 
receiving special education services. In order to maintain the progress that has 
been made in establishing a viable partnership between general education and 
special education, policy in special education must be developed along with that in 
general education, and it must be developed in a way that is maximally integrated 
with general education. If students receiving special education services are not 
being evaluated on the same or a complementary set of outcomes, then educators and 
the general public may not see the value of the participation of these students in 
general education settings. 

The National Center on Educational Outcomes (NCEO) was established in October 
1990 to work with states and other policy groups to develop a model of educational 
outcomes for students with disabilities and to generate a list of indicators of these 
outcomes. The model and list of indicators are to evolve over time as input is received 
from key stakeholders and others interested in assessing educational outcomes. This 
paper represents a preliminary statement of these issues that will likely be modified 
through extensive consultation with policy makers, education evaluators, educational 
researchers, parents, persons with disabilities, and persons from diverse racial, 
ethnic, and linguistic backgrounds. 
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NCEO Approach to Identifying Outcomes 



A brief overview of the NCEO approach to identifying outcomes helps put this 
paper in perspective. First, this paper places the discussion of outcomes and 
indicators within the broader context of education. Within this context, decisions can 
be made about the management of important resources and educational processes to 
achieve the desired outcomes of education. Second, CAs paper begins with 
definitions to clarify the meaning and usefulness of outcomes and indicators with 
which to evaluate the results of education for children and youth with disabilities. 
Third, this paper is portrayed within the fuller NCEO development process for 
designing a conceptual model of important educational outcomes and indicators. 
Each of these approaches and the reasons for them are discussed in brief here so that 
the reader can best understand the three critical sections of this paper (definitions 
of key terms, fundamental assumptions, and the preliminary model of outcomes). 

A Conceptual Model of Education 

The primary value of emphasizing outcomes and indicators in education is to 
improve decisions and the value of educational experiences for students. This 
approach assumes that such decisions can be made more effectively if they are based 
on data. We want warning signs when something can be improved, when something 
is not working right, when education is not producing expected outcomes. Not only 
do we want the warning signs, but we want them to give us an idea of the problems 
and potential solutions within the educational system. And, indicators can 
demonstrate the positive effects of education improvement efforts. 

The attainment of desired outcomes from education is a consequence of many 
factors. Interpreting information on outcomes depends on a full understanding of 
how essential educational resources (contextual and financial) and processes interact 
to influence the results of education (Chen, 1990; DcStefano & Wagner, 1990; Levine & 
Lezotte, 1991; Oakes, 1986; Selden, 1990; Shavclson, McDonnell, & Oakcs, 1989; Walbcrg, 
1984). Understanding the interactions and relationships depends on the results of 
previous, current, and future research and development efforts* Without such 
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understandings, however, policy makers, citizens, and educators are quite limited in 
the options they can exercise to improve the experiences and outcomes of education. 
To be most useful, a conceptual model of education must apply to all students, and 
point to central factors that can be used to alter and improve educational 
experiences. Individual characteristics of a student may be one component of the 
model, but it should be emphasized that the most desirable models are ones that 
effectively include inputs and outputs relevant to all students including those with 
socioeconomic disadvantage, disabilities, or characteristics associated with negative 
educational outcomes. 

Figure 1 is a template for a conceptual model of the resources and processes 
that can influence the attainment of r chool and postschool outcomes for children and 
youth. Important school and longer-term life outcomes are influenced by the 
presence of certain context factors and the availability and application of resources. 
These in turn affect the organization and processes of educational programs and, 
together with educational processes, influence a broad range of educational outcomes 
for children and youth. In Figure 1, some potential context factors, educational 
resources, and educational processes have been listed that interact in a variety of 
ways to affect student attainment of the desired outcomes of education. Such 
influences are important to consider in a model that stresses outcomes of education, 
since such factors are those that can be directly and indirectly influenced to improve 
the results of education. At this point, note that the outcome cells in Figure 1 are 
empty. They will be filled after clarification of developed definitions, assumptions, 
and a suggested model of outcomes* At the same time, we have used two three 
dimensional models to show that outcomes will be assessed in a number of domains, 
direct and indirect outcomes will be assessed, some will be intended outcomes, while 
others will be unintended, and outcomes may be measured before, during and post- 
school* It is important to define and evaluate the relationships that may exist 
between resources and processes, and educational outcomes to help focus decision 
making on the key factors that will improve the appropriateness and effectiveness of 
educational programs and, ultimately, the quality of life for individuals with 
disabilities. 

NCEO Development Process 

The development of a model of outcomes and a list of possible indicators for 
those outcome* is one of six interrelated NCEO activities, A representation of the six 
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Figure 1. Template for a Model of Education [adapted from DeStefano & Wagner, (1991), Levine & Lezotte (1991), 
Oakes <1986), Shavelson (1987)] 
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activities and the relationships among them is provided in Figure 2. This figure 
indicates that the activities are interrelated in that they each affect the others, and 
therefore the development of a model of outcomes and list of indicators is influenced 
by other NCEO activities, such as the ongoing survey of states. Similarly, Strategic 
Planning (Activity 5) and Exchange of Information about the model and indicators. 
Solutions to Technical Issues (Activity 4), and results of analyses (Activity 6) arc 
support and umbrella activities for the other four Center activities. Model 
development and identification of indicators (Activity 2) form the "hub" for all other 
activities. Figure 2 also indicates that the six interrelated activities do not necessarily 
take place in sequential order. Certain activities that initially occur following others 
may cycle back to influence the others. This type of interactive process will 
continue throughout the development and testing of feasibility of a system of 
indicators. For example, state perspectives obtained through the annual surveys 
(Activity 1) will help shape the model of outcomes and the system of indicators 
(Activity 2), the allocation of effort to resolve technical issues (Activity 4), 
information exchange activities (Activity 3) t and other activities. Thus, the 
development and refinement of a model of outcomes and corresponding system of 
indicators is viewed as an ongoing task of the NCEO. 

Model development is depicted in Figure 3. The procedural approach begins 
with developing and gaining consensus on (1) definitions of essential concepts (e.g., 
outcome), and (2) assumptions of the model. Based on consensus about these 
definitions and assumptions, conceptual categories of outcomes, with appropriate 
subcomponents, will be proposed. It is this outcomes model that is used to generate 
indicators of outcomes. All aspects of this initial model of outcomes will be the topic 
of the consensus process. The indicators, in turn, will be refined on the basis of state 
practice surveys, secondary data analysis, identification and solution of technical 
problems, syntheses of research, and the consensus process, 
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Figure 2. Representation of Relationships Among NCEO Activities 
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Figure 3. Representation of Steps in Developing the Model of Outcomes 
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Definitions of Key Terms 



Educators and policy makers speak regularly about outcomes* outcome-based 
education, educational indicators, and outcomes indicators. Nationally there is a shift 
in conceptions of education, away from a focus on issues of access and process in 
providing services toward a focus on the achievement of results or outcomes from 
such services. Increasingly, education is viewed as "the result achieved, the 
learning that takes root when the process has been effective" (Finn, 1990, p. 586). 
Despite the emphasis on achieving results in educational programs, there is little or 
no consensus on exactly what those results should be. or even on the meaning of 
commonly-used terms such as "outcomes" and "indicators." Selden (1990) indicates 
that arriving at agreement about terminology is a difficult process. He states that 
"Just defining indicators and getting everyone in the system, from local school staff 
to national statisticians, to report the figures in a valid and consistent manner, is 
difficult" (p. 383). Selden encourages educators to keep the development and use of 
outcomes indicators simple. Certainly it is important to make sure that everyone is 
starting with the same idea of what terms mean whenever a model of outcomes and 
indicators is proposed. 

The key terms for NCEO's work are "outcomes," "indicators," and 
"comprehensive system of indicators." In this section, the three key terms are 
defined, important points about them clarified, and desired characteristics delineated. 

Outcomes 

In dictionaries, outcomes are defined as results, effects, or consequences of 
events, processes, or experiences. NCEO is using a narrower definition to reflect the 
focus on educational outcomes. The NCEO working definition is as follows: 

Educational Outcome = the result of interactions 
between individuals and schooling experiences. 
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Points of Clarification 

Educational outcomes may be direct or indirect, they may be positive or 
negative, and they may be intended or unintended. There are four key terms in the 
NCEO definition of "outcome": educational experiences, individuals, interactions, and 
result. 

Hie term educational experiences is used to refer to learning experiences that 
are planned, managed, and organized by schools and other agencies designated to 
serve infants, children, and youth. The planning, management, and organizing of 
learning experiences can occur in a number of ways, directly or indirectly, and at 
various levels. For example, teachers plan, manage, and organize activities designed 
to teach reading, language and writing skills, or to facilitate development of positive 
interpersonal relationships among students, improved social adjustment, etc. A 
community agency may develop an Individual Family Service Plan designed to have 
an effect on families and in the long run on individual children. They may become 
significantly involved in the development of individual family service plans for 
infants with disabilities. Learning takes place in schools; it also takes place in non- 
school settings. A teacher may take a class on a field trip to a zoo, based on the belief 
that such trips enable individual children to gain a richer understanding and 
comprehension of a variety of outcomes. 

The term individuals includes infants, children, youth, and young adults. This 
term is used to emphasize the importance of the individual, the critical importance of 
the development and adjustment of individuals in the educational system, and the 
growing recognition that education is a life-long learning process. At the same time, 
the plural form is used to reflect the interest in group-level information needed for 
decision making at the state level and even the national level. 

The term interactions is used to recognize the fact that, while schools bring to 
students a set of experiences, individual students, including those with disabilities, 
differ in the experiential background, characteristics, and history with which they 
enter learning situations. They also differ on a set of characteristics and 
competencies important to the process of learning, including their skills, abilities, 
temperament, and motivation to learn. These individuals attend schools and engage 
in life experiences where they learn and acquire skills. The product of the 
recurring interactions between the individual and school experiences, which in 
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turn are influenced by the individual's life experiences, is an educational outcome. 
The interaction is reciprocal between the individual and the planning, management, 
and organizing of learning experiences. 

Learning experiences are planned, managed, and organized for the purpose of 
bringing about ifisulu or effects. Educators ard developmentaiists commonly refer 
to results such as achievement, graduation, postsecondary status (employment, 
education), and life adjustment. While the intended outcomes of educational 
experiences arc positive results, it is recognized that to obtain a true picture of the 
health of the educational system, some negative unintended outcomes may need to be 
monitored as well. 

DfiSired — Characteristics nf OutrnmP* 

Educational outcomes are the results of interactions between individuals 
and educational experiences. We believe that the outcomes relevant for our 
purposes should: 

• Be valued by society; 

• Reflect the degree to which essential cultural expectations have 
been attained in such areas as participation and' achievement in 
normal social, educational, training, employment, leisure, and 
community environments; 

• Include both direct and indirect results of educational experiences; 

Be relevant to all individuals, regardless of personal characteristics; 
and 

Be a product of learning and experience (what has been learned) 
rather than how learning occurred. 

Indicators 

In dictionaries the term indicator is defined as a number, index, feature, or 
measure that enables comparisons to be made- NCEO is using a broader definition to 
reflect the focus on both quantitative and qualitative indicators. The NCEO working 
definition is as follows: 
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Indicate = a symbolic representation of one or 
more outcomes (or inputs, contexts, or processes) 
that can be used in making comparisons. 



Hie word indicator carries multiple meanings. The general public thinks of 
economic indicators like the Consumer Price Index, the rate of unemployment, and 
the Gross National Product. Educational policy makers often talk about input, process, 
context, and outcome indicators. While NCEO's immed'ite focus will be on outcome 
indicators, other indicators (those for inputs, contexts, and processes) are 
important in the conduct of education, and should be defined to assist in improving 
the management and operation of responsible educational decision making and 
programs. 

Points of riflriflcuHon 

The important terms in the definition of "indicator" are symbolic 
representation, outcomes, and comparisons. 

The term symbolic representation is used in this definition rather than the 
term "number" because indicators can be other than numerical. Indicators may be 
quantitative (percentage of students who . . . , scores on an achievement test), or they 
vr j be qualitative (degree of satisfaction with , . . , appropriateness of organization 
and delivery of instruction). This definition recognizes that some properties of 
educational systems are not directly or perfectly measurable and representable by a 
specific number (Shavelson, McDonnell, & Oakes, 1989). 

Outcomes, as noted previously, are the results of interactions between 
individuals (infants, children, and youth) and educational experiences (which in the 
case of infants may include early interventions). Although NCEO is beginning with a 
focus on indicators of outcome, it is important to keep the broader perspective in 
mind. Indicators of input, context, and process are also important to assess when 
modifying and managing policies and practices to improve educational experiences 
and outcomes for students with or without disabilities. 

Comparisons is the third key term in the definition. Comparisons arc made at 
different levels, at different times, and using differing referents (Smith, 1988). Some 
comparisons of indicators involve comparisons at time 1 and time 2. Others involve 
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comparisons of differences among outcomes at a specific time. Some comparisons 
involve looking at outcomes in reference to a standard (like the national goals). 
Selden (1990) stated that: 

Analyzed over time, used to compare units, or examined in relation to a 
social goal or standard, indicators . . .(reveal trends, show where relative 



in relation to how we want to do." (p. 384) 

Desired Qiaiaslgil&llM of Indicators 

Indicators are symbolic representations that enable comparisons to be made 
of one or more educational outcomes for infants, children, and youth. They shouIJ: 

• Be reliable and valid proxies of the desirable outcomes of education 

(e.g., time in school, achievement, etc.); ^ 

v 

• Be directly associated with particularly significant aspects of 
schooling or areas of policy interest; 

• Be expressed in a positive direction to the extent possible, but also be 
sensitive to unintended effects; 

• Include both individual-level and system-level representations of 
outcomes; 

• Be based on procedures that are concise, timely, reliable, and valid; 
and 

• Use procedures that are already available, or feasible to derive from 
existing data, if possible. 



NCEO is using a broad definition of a comprehensive system of indicators (CS1) 
even though its initial focus will be on developing a system of outcomes indicators. 
The NCEO working definition is as follows: 



strengths 




report how we are doing 



Comprehensive System of Indicators 



Comprehensive System of Indicators (CS1) = a 



set of indicators of educational inputs, contexts, 



processes, and outcomes. 
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Points of Clarification 

Although the term "comprehensive" is used to describe the system of 
indicators, it is recognized th*? probably no identified system of indicators is truly 
complete. The system of indicators to be identified by NCEO is seen as evolving over 
time. This will always be so as the educational environment changes. The main 
terms in the definition of a comprehensive system of indicators (outcomes and 
indicators) have been defined previously. New terms used in this definition are 
inputs, contexts, and processes. 

Inputs are financial and human resources that are available to the schools 
and the educational system for use in meeting stated missions and objectives. Inputs 
include fiscal resources, teacher quality, and the characteristics of students who arc 
served. 

Contexts are characteristics of schools, and characteristics of the 
communities in which students live and develop. Community socioeconomic status 
(SES), for example, is among the most frequently identified contextual aspects of 
education. 

Processes are events such as "the adequacy of the curriculum and 
instruction received by students, the nature of the school as an organization in 
pursuit of educational excellence and equity" (Shavelson et ah, 1989, p. 7). These 
include the types of characteristics identified in research on critical instructional 
factors and effective schools research (Christenson, Ysseldyke, & Tburlow, 1989; 
Levine & Lczottc, 1991; Walberg, 1984). 

He&llXd PharactgrUtlgg nf a Comprehensive System of Indicators 

A comprehensive system of indicators is a set of indicators of inputs, 
contexts, processes, and outcome! A comprehensive system of indicators should: 

• Show the relationships that exist or are hypothesized to exist among 
inputs, contexts, processes, and outcomes; 

• Include quantitative and qualitative data; 

• Include both individual-level and system-level indicators; and 

Provide data defined in terms of time (when data elements are 
collected) that are relevant to current or anticipated problems 
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Fundamental Assumptions Guiding NCEO Activities 

Before identifying a model of outcomes and a system of indicators for students 
with disabilities, it is important to articulate a perspective on special education and 
related services. Implicit in the model is the view that special education is a 
capacity-building resource for general education and a support to students and 
families. The purpose of special education is to enable each individual with a 
disability to live as normal a life as possible. Thus, special education exists for the 
purpose of preventing impairments from becoming disabilities and for the purpose 
of compensating for disabilities* Included in our schools today are students with 
visual, hearing, emotional, learning, physical, and multiple disabilities. Yet, policy 
makers who have been identifying goals and assessing outcomes for students in 
America's public schools often have omitted from consideration these students with 
disabilities. 

The foundation for a comprehensive system of indicators (CSI) rests upon a 
number of critical underlying assumptions and premises, each of which has 
implications for the development of the CSI. Presented in this section arc the critical 
assumptions or premises that guide NCEO activities. They are organized by the key 
concept to which they relate; the model of outcomes, the specific outcomes 
indicators, or the comprehensive system of indicators (CSI). 

Model of Outcomes 

Assumption 



Assumption #1 

A model of outcomes is needed for all students, and at the broadest 
level, should apply to all students regardless of the 
characteristics of individuals. 
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Implication: The model of outcomes developed must be comprehensive, 
including the range of outcomes desired for all students, including those in special 
education. However the relative emphasis on outcomes in education may vary for 
children and youth with particular characteristics. For some, we may demand 
attainment of only the cultural imperatives; for others we may demand mastery of 
calculus. This may result in specific indicators of outcomes being given differing 
emphasis for students depending on their individual characteristics. 

Outcome Indicators 

Assumptions 



Assumption £2 

Indicators of outcomes for students receiving special education 
services should be related, conceptually and statistically, to those 
identified for students without disabilities. 



Implication: Indicators will be either general or specific, with most applying 
broadly across a range of characteristics (age levels, severity of disability, or types of 
disability) and some being specific to given levels of development, severity of 
disability, or types of disability. The most useful indicators will be those that apply to 
both general education and special education. Indicators also can be very specific, 
such as indicators that are relevant only to those students planning to enter college 
(e.g., SAT and ACT scores used in Wall Chan). 



Assumption £2 

Indicators should be unbiased with respect to gender, culture, 
race, and other characteristics of the diversity of students in 

today's school population. 
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Implication: Today's schools include students from diverse backgrounds. 
Indicators of outcomes must be a fair representation of the accomplishments of all, 
and understandable to multiple audiences and constituencies. Indicators for students 
with specific impairments related to methods of data collection should appropriately 
represent the students' attainment and not be affected by response-related 
impairments. 



Assumption #4 
While indicators should meet research standards, those that 

do not could still be used. 



Im plic a t i on - Indicators can be identified and defined in a way that would meet 
research standards. However not all data collected for an indicator may have these 
ideal characteristics. For some purposes, it is necessary to assess the effects and 
delineate the extent to which dau can be less than optimal, yet included in the 
measurement of an indicator. 



Comprehensive System of Indicators 

Assumptions 



Assumption #S 
A comprehensive system of indicators should provide data 
needed to make policy decisions at the state and 

national levels. 



Implication : Indicators arc useful for policy when they represent variables 
that can be influenced by policy and management practices. Each indicator must be 
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useful in a policy context at a specific level. NCEO is initially concerned primarily 
with indicators for the state and national levels. Although an indicator that is useful 
at these levels may not be directly useful at the local level, a comprehensive model of 
indicators should be helpful at all levels of education in suggesting areas of 
importance for structuring learning and evaluation systems at district, school, and 
classroom levels. It is beneficial to delineate how indicators at different levels 

relate to each other. The application of definitions, concepts, and outcome areas will 
increasingly involve further application in local districts and classrooms. 



Assumption #6 
A comprehensive system of indicators should to the maximum 
extent possible be based on demonstrated functional relationships 
between outcome indicators and indicators of educational inputs, 

contextual characteristics, and processes; however, valued 
indicators may be included even if functional relationships have 

not been established. 



Im plicati o n ,: A system of outcomes indicators should be part of the broader, 
evolving comprehensive picture of the educational system, including input, context, 
and process indicators. The education system exerts its primary influence on the 
management of resources and conditions of education. Assessment of educational 
outcomes, therefore, is most useful when it is linked to decisions that can be made to 
improve educational experiences for students. While the current and primary focus 
of NCEO is on developing a methodology to assess outcome indicators, attempts will be 
made to place these indicators within the broader comprehensive system of 
indicators. Specific efforts directed to special education outcome indicators must be 
based on existing special education research and on general education research on 
academic and related attainments. 
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Assiimntinn £2 

A comprehensive system of indicators should be flexible, 
dynamic, and responsive to review and criticism, changing 

to meet identified needs and future developments in the 
measurement of inputs, contexts, processes, and outcomes. 



Implication Identified indicators must be relevant to current circumstances 
and stakeholder values. When circumstances change, it must be recognized that 
there may be a need to change the system of indicators. The system must provide data 
relevant to current or anticipated problems and issues. 
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A Preliminary Model of Outcomes 



Two possible models of outcomes are suggested in this section. They represent 
alternative ways of thinking about outcome areas and their relative importance. 
This thinking reflects the need repeatedly expressed by special educators for our 
educational system to go beyond assessment of only academic achievement, 
particularly if it is to be truly inclusive of ail of America's students. 

Model 1 

One set of broad outcome areas and an initial breakdown that might be 
included in a model of outcomes are shown in Figures 4 and 5. In the model 
illustrated in Figure 4 we have identified five basic outcome areas: 




Presence in the System 
Participation In the System 
Accomplishment 
Contribution 
Satisfaction 



The outcome areas represent the foundation of the model of outcomes. The need for 
each of the five outcome areas is described in brief here, along with some examples 
of educational goals that are encompassed in each. 

Presence in the system refers to opportunities for physical integration of 
students in schools and communities. While access to education is now recognized as 
a right of all students, this right has existed for students with disabilities only since 
1975. Even today, the meaning of "presence in the system" varies. Thus, it is 
important not to lose sight of the need for students with disabilities to be included to 
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the maximum extent possible in the general education system. A reasonable goal for 
our nation is to increase the number of students receiving all or part of their 
education in regular education classes. Beyond school, a reasonable goal is to 
increase the number of people with disabilities who are gainfully employed. 
Hie area of Participation addresses the extent to which persons with 
disabilities experience opportunities for active involvement in schools, social 
experiences and relationships, community participation, meaningful work and other 
essential areas of living. We know that a relationship exists for all individuals 
between their involvement in an activity and their commitment to the activity. 
Students who drop out of school tend to be those who have not been involved in 
extracurricular activities, those who have become alienated from the educational 
environment. We know that students who actively participate in their learning 
perform better than students who do not demonstrate active participation. Likewise, 
we know that individuals with disabilities show dramatic improvement in their 
adaptive behavior and self-help skills when they participate with peers who do not 
have disabilities. Participation is a critical aspect of our educational system. An 
important goal for our nation is to increase the number of ways in which students 
with disabilities participate in schooling events. For those beyond school, a goal is to 
increase the number of community events in which people with disabilities 
participate. 

Accomplishment is perhaps the broadest and most complex area to define. It 
includes the skills and competencies considered important to the development of 
personal independence and productivity in family, social community and economic 
circumstances. Accomplishment can be academic in nature, but it also can be 
physical, attitudinal, or emotional. Academic accomplishments are the focus of most 
current goals. Two of our national goals relate most directly to this area of 
accomplishment: 

Goal 2: To increase the Ligh school graduation rate to at least 90 
percent by the year 2000 

Goal 3: To have American students leaving grades four, eight, and 
twelve having demonstrated competency in challenging subject matter 
including English, mathematics, science, history, and geography; and 
every school in America will ensure that all students learn to use their 
minds well, so they may be prepared for responsible citizenship, 
further learning, and productive employment in our modern economy. 
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It is equally important for all students with disabilities receiving education in 
regular classes to demonstrate progress along with their classmates. And, it is 
important for students with disabilities receiving education in special classes to 
demonstrate progress in academic and other areas. 

The area of Contribution comprises a range of outcomes that describe the 
impact of the person in schools and communities. This area reflects the notion that 
individuals should give something back to society for the educational and other 
benefit provided. Common examples in this area include volunteerism, paying taxes, 
and not committing crimes. During the school years, this also could be interpreted as 
including serving on student committees, keeping instructional materials in good 
condition, and not defacing school property. Many of these same goals can be 
defined for individuals with disabilities. We want them to be contributing members 
to our society. 

Finally, the proposed model includes the area of Satisfaction to suggest that 
an important aspect of outcomes is the level of personal and broad community 
satisfaction with the results of education. We strive for satisfaction in more than just 
the students with the educational process,. We also want the satisfaction of educators 
and future employers. Much of the impetus for continued educational reform has 
come from the business community and its realization that schools are not producing 
individuals with the skills necessary to assume jobs in the workforce. Satisfaction 
goals that we now have apply to individuals with disabilities. We want these students 
and their parents to be satisfied with their education. We also want those who 
interact with them in the future, particularly employers, to be satisfied with the 
skills that they bring to the job. 

The five broad outcome areas of presence, participation, 
accomplishment, contribution, and satisfaction can be organized into a 
model of outcomes similar to that shown in Figure 5. The five outcome areas 
represent the foundation of the model. Each of the areas can next be further divided 
into domains, some of which are school outcomes and others of which are postschool 
outcomes. For example, accomplishment can be divided into domains of 
socioemotional, health and physical competence, cognitive/conceptual, attitudes, 
adaptive living, and possibly others. Each of the domains can be further divided into 
subdomains. For example, cognitive/conceptual outcomes could include quantitative, 
writing, communication (literacy/reading), language, critical thinking, and 
probably others. 
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Figure 5. Conceptual Model of Outcome Areas, Domains, Subdomains, and Indicators 
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Model 2 

While the importance of the five outcome areas is clear, the suggested equal 
value of some of the five outcomes, presence and participation for example, to 
accomplishment has been questioned. Currently, major education policy groups 
focus almost entirely on accomplishments. Presence and participation are 
mentioned almost exclusively in relation to graduation rate and dropout rate. Only 
rarely are satisfaction and contribution (such as volunteerism) included. 

An alternative model to one showing five broad outcomes is illustrated in 
Figure 6. Here all outcomes are viewed as accomplishments, and accomplishments 
are evaluated in each of nine basic outcome domains within Accomplishments: 



Preliminary Model 2: Outcome Domains 
in Accomplishments 



Cognitive and Conceptual 
Affective and Emotional 
Attitudes 
Physical and Health 
Contribution 
Satisfaction 
Adaptive Living 
Participation 
Presence 



Each of these domains would then be further subdivided. For example, 
cognitive/conceptual could be divided into the subdomains of literacy, writing, 
quantitative, language, critical thinking, and others. Participation couid be divided 
into subdomains such as extracurricular activities, appropriate leisure activities, 
post-secondary education, and others. 
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A System of Indicators 



The long range work of the NCEO will focus upon refinement and agreement 
on components of a conceptual model and a Comprehensive System of Indicators to 
describe and evaluate educational experiences for children, youth and adults with 
disabilities. 

When the model shown in Figure 4 or 6 is completely specified, it will be 
possible to list specific indicators that can be used to assess outcomes for evaluative 
and decision-making purposes. In the table below we have listed an illustrative set of 
indicators for several outcomes areas. These were chosen as illustrative only, and 
are viewed as springboards for discussion. The listing of indicators highlights 
several issues in assessment of outcomes for students with disabilities. These 
unresolved issues are discussed in the next section. 



Possible 


Outcomes and Indicators for 


Selected Domains 


Outcome Domain 


Outcome 


Indicator 


Presence 


Students attend school 


• Dropout rate 

• Suspension rate 


Pfcysical/Healih 


Normal growth and 
development 


• Proportion of children in 
expected range 




Adequately nourished to 
participate in school 


« Nutritional status of 
children 


A ffectivr/Emotional 


Absence of crime 


• Violent deaths 



* Juvenile crime 



Satisfaction 



Employers and post- 
secondary educators 
arc satisfied with the 
products of the 
educational system 



Number or proportion of 
employers and post- 
secondary educators satis- 
fied with the products of 
the educational system 
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Possible Outcomes and Indicators for Selected Domains — continued 



Outcome Domain 


Outcome 




Indicator 


Contribution 


Makes a 


positive 




• Number of people who pay 




contribution to 


family, 


taaes 




school, 


and 




• Number of people who rely 




community 




on social support networks 


Cognitive/Conceptual 


Reads at 


proficient 


level 


• Percentages of students at 










proficient level or NAEP 










reading assessment 



Many of the outcomes and indicators in this table are derived from materials prepared for an 
"Outcomes Group Meeting" (April 17-18, 1991) of Joining Forces, a joint project of the 
American Public Welfare Association (APWA) and the Council of Chief State School Officers 
(CCSSO). 
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Unresolved Issues 



In considering possible outcomes for children and youth with disabilities 
several issues must be addressed. Decisions about these issues will directly affect the 
nature of the outcomes model and the indicators that are identified. Those who read 
this paper are encouraged to submit to us their thoughts on the issues. 

Educators and policy makers must decide specifically the outcome areas in 
which data will be collected on students with disabilities. Currently there is 
considerable variability in the kinds of data that state departments of education 
collect, and there is little consistency in the outcome domains in which data are 
collected. For the most part, those states that do collect outcome data gather data on 
s;-dcnt academic accomplishments (reading, math, etc). Data are collected using 
published instruments or state mastery tests. It is common for students with 
disabilities to be excluded from statewide testing programs. We think the following 
kinds of issues will have to be resolved before decisions can be made on the specific 
kinds of indicators to be used. 

Intended Versus Unintended Outcomes 

One of the broad issues to be address is whether to collect information only on 
those outcomes that were intended, or to also look at unintended outcomes. For 
example, having an effect on the emotional well-being of an individual may be an 
unintended outcome of increasing educational standards. Yet, it could be argued that 
if the imposition of new standards increases the suicide rate among adolescents, it 
becomes an important educational outcome about which we need to know. There are 
many potential unintended outcomes of education. The extent to which these can be 
identified and, if identified, used in a model of outcomes for education will require 
considerable discussion. 

Direct Versus indirect Oiitrnmps 

One of the broad issues to be addressed is the directness of outcomes in which 
we are interested. We can talk about direct outcomes, or we can talk about indirect 
outcomes, or we can talk about both. Direct outcomes are those outcomes for which 
schools can be held directly accountable, such as a student being able to read at a 
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cenain level or a student being able to tell time. Indirect outcomes are those that are 
less obviously tied to the school curriculum, such as a student being gainfully 
employed upon graduation. Most often, direct outcomes are those that a student 
should be able to demonstrate at various points during and at the end of a school 
career. Indirect outcomes more often are those that a student would demonstrate 
after finishing school, but not always. The distinction remains that some outcomes 
are more directly linked to school programs than others. At some point we must 
make a decision about how closely linked our identified outcomes must be to the 
educational programs we provide. The primary focus in our work is on direct 
outcomes. Yet, research obviously is needed and should be pursued in the "indirect 
outcomes" area, since direct outcomes (what schools teach) are influenced by many 
factors before resulting in the indirect outcomes, particularly as severity of 
disability increases. 

Same Versus Different Indicators 

A decision must be made on the extent to which indicators for students with 
disabilities should differ from indicators for students who are not disabled. We have 
made the assumption that indicators of outcomes for students receiving special 
education services should be related, conceptually and statistically, to those identified 
for students without disabilities. Does this mean that special education will or will not 
have a separate set of indicators? There are two prevailing views on this matter. 
Many believe that the indicators used in special and general education ought to be 
identical. They believe that the same kinds of data should be gathered on all students. 
For others, it makes little sense to expect that students with disabilities, especially 
those with very severe disabilities, can or should participate in the assessment 
system used to gather outcomes data on students who are not disabled. It is argued 
that "different" kinds of data will have to be gathered on students with disabilities, 
and that a separate assessment system will have to be developed. 

Category Snecific Versus Nnncategorical Indicators 

A decision must be made about whether different kinds of data will be collected 
as a function of student characteristics. We have assumed that "A mode! of outcomes 
is needed for all students, and at the broadest level, should apply to all students 
regardless of the characteristics of individuals. " We assume that the same kinds of 
data would be gathered on the different categories of students. Others take issue with 
this assumption. They argue that category-specific indicators are needed, since 
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students in the various categories have unique needs. Such a position is reflected in 
data collection efforts that are category specific. 

Indicators Differentiated hv Severity of Disability 

A decision must be made about whether outcomes differ as a function of 
severity of disability. Will different kinds of data be gathered on students with 
severe impairments in contrast to those with mild impairments, perhaps ignoring 
the specific category of disability? Or, will we gather the same kinds of data, but 
expect different levels of accomplishment? 

Indicators Differentiated by Developmental Level 

A decision must be made about whether outcomes differ at differing 
developmental levels. Do we gather the same kinds of data on infants, at school 
entrance, during school, and post-school? Or are different kinds of data gathered at 
these different developmental levels, or at some subset of them? 

System Level Versus Individual Level Indicators 

We must decide whether the system of indicators we develop should be 
indicators at the system level, the level of the individual, or some mixture of the two. 
Some outcomes are system-level outcomes while others are individual-level outcomes. 
For example, when we talk about the percentage of students graduating from school, 
or the percentage of students passing a minimal competency test, we are talking 
about system-level outcomes. When we talk about the average reading score of tenth 
graders or a student's degree of involvement in volunteer activities, we are talking 
about individual-level outcomes. Both of these levels have been used in most 
discussions of outcomes. Whether these levels should remain together, be separated, 
or be addressed through some other option, will be debated. 
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Summary 



In response to the reed for educational policy to be truly inclusive of all » 
students, and to improve educational results for students with disabilities, NCEO has 
been established to promote the development of a comprehensive system of 
educational outcome indicators for infants, children, and youth with disabilities. The 
Center also seeks to promote national discussion of educational goals and indicators of 
educational outcomes that arc inclusive of students with disabilities. The Center is 
fulfilling its mission through development of a model of outcomes, identification of 
indicators, descriptions of state practices, analysis of available data, development of 
solutions to technical issues, and dissemination of information. 

This paper is a description of some initial efforts to define outcomes and 
indicators, develop operating assumptions, and illustrate a tentative model of 
outcomes and the development of indicators from it. Further work will be 
undertaken to refine these concepts, through an extensive development and - 
consensus review process, and the eventual development of a comprehensive 
compendium of outcomes and outcome indicators. This effort will lead further to the 
resolution of technical issues, technical assistance activities, and national data from 
available sources, on educational status and outcomes of children and youth with 
disabilities. 

The consensus process will be directed toward a number of goals, with the 
emphasis changing over time. Awareness building, exchange of information, and 
formalizing relationships are among the major targeted goals for the the Center's 
first two years. As these goals are met, greater emphasis will be placed on 
influencing and convincing, mobilizing regional and state discussions, and creating 
dialog among states, national policy groups, and federal groups. The end goal of the 
consensus process is to achieve agreement, and shared vision, and cooperation 
within and among states and federal groups. Strategies for the consensus building 
process include structured meetings (using nominal group procedures, computerized 
systems, and learning and development approaches.), mail and phone connections, 
and a computerized communication network. The consensus process also will be 
strengthened through the participation of Center staff in other groups' activities and 
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through presentations and information exchanges at conferences. Reactions to this 
paper represent one very important step in the consensus building process. Some 
important questions on which input is desired include: 

1. Do the outcome areas defined in the preliminary model (either Model 
1 or Model 2) include all that should be covered and that apply to all 
students? 

2. To what extent do the listed Assumptions seem appropriate as we 
begin to address the issue of constructing a conceptual model of 
outcomes and a comprehensive list of indicators? 

3. What is your opinion of each of the issues identified in the 
Unresolved Issues section? 

Intended vs Unintended Outcomes 

Direct vs Indirect Outcomes 

Same vs Different Indicators 

Category Specific vs Noncategorical indicators 

Indicators Differentiated by Severity of Disability 

Indicators Differentiated by Developmental Level 

System Level vs Individual Level Indicators 

4. Are there additional issues that need to be addressed? 

5. Part of the Center's purpose is to engage in consensus-building 
around the model of outcomes and the comprehensive system of 
indicators. Is this an achievable goal? What can we do to maximize 
the possibility of building consensus? 

Please call Jim Ysseldyke (612-624-0414) or Martha Thurlow (612-624-4826) with 

comments, or send FAX to 612-624-9344 or 612-624-0879, or mail them to: 

National Center on Educational Outcomes 
111 Pattee Hall 
ISO Pillsbury Drive SE 
Minneapolis, MN 55455 

We look forward to your input. 



9 

ERIC 



29 

42 



NCEO Draft Working Paper 
7/23/91 



References 

Chen, H. (1990). Issues in constructing program theory. In L. Bickman (Ed.) New 
directions for prog ram evaluation (Volume 47). San Francisco: Josscy-Bass. 

Christenson, S.L., Ysseldyke, J.E., & Thurlow, M.L. (1989). Critical instructional 

factors for students with mild handicaps: An integrative review. Remedial 

and Special Education 10. 21-31. 

DeStefano. L. & Wagner, M. (1990). Outcome assessment in special education: Lessons 
learned . Transition Institute at Illinois, University of Illinois at Urbana- 
Champaign, and SRI International, Menlo Park, CA. 

Finn, C E (1990). The biggest reform of all. Phi Delta Kapp an. J\. 584-592. 

Levine, D. V., & Lezotte, L. W. (1991). Unusually effective schools: A review and 

analysis of research and nractice. Madison,. WI: National Center for Effective 
Schools Research and Development. 

National Council on Disability (1989). The education of students with disabilities: 
Where do we stand. Washington, DC: Author. 

♦ . 

Oakes, J. (1986). Educational indicators: A guide for policymakers. Santa Monica, CA: 

RAND Corporation, Center for Policy Research in Education. 

Selden, R. W. (1990). State indicator systems in education . Washington, DC: Council 
of Chief State School Officers. 

Shavelson. R., McDonnell, L., & Oakes, J. (1989). Indicators for monitoring 

mathematics and science education. Santa Monica, CA: RAND Corporation. 

Smith, M. S. (1988). Educational Indicators. Phi Delta Kappan . 69. 487-491. 

Walberg, H. J. (1984). Improving the productivity of America's schools. Educational 
Leadership . 41(81. 19-30. 



30 

13 



NCEO Draft WwWng Paper 
7/22/91 



Individuals Providing Reactions to drafts of this Working Paper 



Other NCEO Staff and Consultants 

Bob Algozzine 

MJchalChvat 
Dean Honetsch lager 
David Madaon 
Mary Schaefer 

Amy Spiegel 

Charie Lakln 

State Directors of Special Education 

Unda Hargan, Kentucky 
Mary Beth Fafarti, Massachusetts 
Dean Meyers, South Dakota 
Wayne Erickson, Minnesota 
Patrick Campbeil, Caiomia 
Tom GUfiung, Connecticut 
Steve KuHc, Utah 

OSERS Reoresantativflg 

Martha Coutinho 
Lou Danielson 
Martin Kaufman 



Members of the Mid-South RRC Evaluation Forum 

Ken Olsen, MSRRC 
Carol Massanart, MSRRC 
Martha Brooks, Delaware 
Larry WexJer, District of Columbia 
Nancy LaCount, Kentucky 
David Haycten, Maryland 
Fred Baare, North Carolina 
Austin Tuning, Virginia 
Bin Frey, Michigan 



Although the individuals listed above provided us with many and varied comments and suggestions 
about this Working Paper, the responsibility for the paper is solely that of the authors. We selectively 
picked among the many ideas presented to us by readers. Thus, the above individuals should not be 
considered to agree with ail of its contents. 



e 

ERIC 



31 

4\ 



